System and method for tying together machine vision coordinate spaces in a guided assembly environment

ABSTRACT

This invention provides a system and method that ties the coordinate spaces at the two locations together during calibration time using features on a runtime workpiece instead of a calibration target. Three possible scenarios are contemplated: wherein the same workpiece features are imaged and identified at both locations; wherein the imaged features of the runtime workpiece differ at each location (with a CAD or measured workpiece rendition available); and wherein the first location containing a motion stage has been calibrated to the motion stage using hand-eye calibration and the second location is hand-eye calibrated to the same motion stage by transferring the runtime part back and forth between locations. Illustratively, the quality of the first two techniques can be improved by running multiple runtime workpieces each with a different pose, extracting and accumulating such features at each location; and then using the accumulated features to tie the two coordinate spaces.

RELATED APPLICATION

This application claims the benefit of co-pending U.S. ProvisionalApplication Ser. No. 62/201,723, entitled SYSTEM AND METHOD FOR TYINGTOGETHER MACHINE VISION COORDINATE SPACES IN A GUIDED ASSEMBLYENVIRONMENT, filed Aug. 6, 2015, the teachings of which are expresslyincorporated in by reference.

FIELD OF THE INVENTION

This invention relates to machine vision systems, and more particularlyto vision systems used to guide the assembly of workpieces and otherobjects in a manufacturing and guided assembly environment.

BACKGROUND OF THE INVENTION

In machine vision systems (also termed herein “vision systems”), one ormore cameras are used to perform vision system process on an object orsurface within an imaged scene. These processes can include inspection,decoding of symbology, alignment and a variety of other automated tasks.More particularly, a vision system can be used to inspect a flat objectpassing through an imaged scene. The scene is typically imaged by one ormore vision system cameras that can include internal or external visionsystem processors that operate associated vision system processes togenerate results. One or more of the cameras are calibrated to enableit/them to perform the vision task(s) with sufficient accuracy andreliability. A calibration plate can be employed to calibrate thecamera(s), and establish a common (global) coordinate space (alsoreferred to as a coordinate “system”) in which the pixel locations ofall cameras are mapped to associated points in the coordinate space,thereby allowing an imaged feature in any camera to be located withinthe coordinate space. In applications that use a motion stage forguiding the assembly of objects the calibration process can includeestablishing a relationship between the motion stage (also termed a“motion coordinate space”) and the common coordinate space. Suchcalibration can be achieved using known “hand-eye” calibrationtechniques.

A notable task for vision systems is to assist in guiding and verifyingthe assembly of objects (also termed “workpieces”) by an automatedassembly mechanism that can include moving platforms (motion stages) toaccurately support a workpiece at an assembly location and a manipulator(e.g. a robot hand or arm that performs a “pick-and-place” operations oranother type of motion device/motion stage) to move another workpiece inan “assembly motion” to an overlying alignment location where it isassembled to the workpiece. A particular pick and place operationinvolves aligning one workpiece with another workpiece. For example, atouch screen can be manipulated by the manipulator over and into a wellon a cell phone residing on the motion stage in the pick-and-placeoperation, where each touch screen is moved from the pick location anddeposited at the place location (sometimes termed a “station”) on awaiting cell phone body. Proper and accurate alignment of the touchscreen with cellphone body is highly desirable.

Some exemplary systems are initially trained so that each workpiece isin correct alignment with the other workpiece during runtime operationof the system. During train time, the workpieces are positioned at theirrespective locations/stations so that, when assembled, the assembledworkpieces have a desired mutual positional relationship. Followingtraining, during runtime, one or both of the workpieces are repositionedin their respective locations by use of the vision system in control ofthe associated motion stage at the location, in order to account for anyplacement or dimensional variation, and then assembled. By adjusting thestage, the workpieces are thereby placed in same mutual (expected)positional relationship as they were at train time. In other exemplarysystems, training can be omitted—for example systems where the geometryof the assembled parts can be employed during runtime, such as where afirst rectangular object is being centered on a second rectangularobject.

Following imaging of the scene with workpieces, features belonging toeach workpiece are extracted and mapped to the common coordinate spaceas part of a process that eventually computes the stage motionparameters that would result in the desired assembly. The mapping to thecommon coordinate space is due to a calibration process. On manyassembly systems, the calibration process includes transferring aspecial calibration target between locations using the repeatablemanipulator that transfers the part and imaging a calibration target atboth the locations. This technique has several disadvantages. Forexample, the pick-and-place gripper that transfers the part may not beable to transfer a calibration target, due to differences in the shape,weight or other characteristics of the workpiece versus the calibrationtarget. That is, a calibration target does not fit the parameters of thegripper. In such systems, the mapping of parameters to the commoncoordinate space must be specified manually via user input, which oftenresults in sub-optimal calibration. In addition, the motion pathfollowed by the pick-and-place manipulator during calibration versusruntime may differ due to differences in the characteristics of the partand the calibration target, such as differences in thickness or othercharacteristics. This can introduce calibration errors as it requiresvariation from the runtime motion steps taken by the manipulator versusthe training time motion steps. More generally, it is a furtherinconvenience for a user to employ a separate calibration target duringcalibration time due to additional setup steps, etc.

Also disadvantageously, prior techniques for calibrating have relied oncertain mechanical or/or iterative trial and error principles that havelimitations and/or are inconvenient and time-consuming to implement. Forexample, in one technique the mechanical system can be constructed sothat the relationship between the coordinate space at the first locationand the coordinate space at the second location is known andestablished. This technique limits flexibility and does not account forpossible movement or change in position over time. In another technique,an initial estimate of the relationship between the two locations ismade and is repeatedly refined by using the quality of the assembledparts. This technique is time consuming and relies upon multipleiterations to achieve desired accuracy.

SUMMARY OF THE INVENTION

This invention overcomes disadvantages of the prior art related toproblems that arise from transferring a calibration target by themanipulator (and associated gripper) between pick and place locations,by providing a system and method for calibration that ties thecoordinate spaces at the two locations together during calibration timeusing features on a runtime workpiece. This system and methodaccommodates at least three different scenarios/techniques—one in whichthe same features can be imaged and identified at both locations; one inwhich the imaged features of the runtime workpiece differ at eachlocation (wherein a CAD or measured rendition of the workpiece isavailable); and one in which the first location containing a motionstage has been calibrated to the motion stage using hand-eye calibrationand the second location is hand-eye calibrated to the same motion stageby transferring the runtime part back and forth between locations.Illustratively, the quality of the first two techniques can be improvedby running multiple runtime workpieces each with a different pose,extracting the features and accumulating such features at each location;and then using the accumulated features to tie the two coordinatespaces. More generally, the system and method independently calibratesthe two locations and ties the coordinate spaces for the two locationstogether by transferring a workpiece that the manipulator is constructedand arranged/adapted to transfer during assembly and using thatworkpiece's features instead of using the features of a calibrationplate.

In an illustrative embodiment, a system and method for calibrating avision system in an environment in which a first workpiece at a firstlocation is transferred by a manipulator to a second location isprovided. An operation is performed on the first workpiece, which reliesupon tying together coordinate spaces of the first location and thesecond location. At least one vision system camera is arranged to imagethe first workpiece when positioned at the first location and to imagethe first workpiece when positioned at the second location. At least onevision system camera is calibrated with respect to the first location toderive first calibration data which defines a first coordinate space andat least one vision system camera (potentially the same camera(s)) iscalibrated with respect to the second location to derive secondcalibration data which defines a second coordinate space. The featuresof at least the first workpiece are identified at the first locationfrom a first image of the first workpiece. Based on the identifiedfeatures in the first image the first workpiece is located with respectto the first coordinate space relative to the first location. The firstworkpiece is gripped and moved, with the manipulator, at least one time,to a predetermined manipulator position at the second location and asecond image of the first workpiece is acquired at the second location.Based upon the identified features in the second image, the firstworkpiece is located with respect to the second coordinate spacerelative to the second location. The first coordinate space and thesecond coordinate space are thereby tied together. Illustratively, wherethe identified features in the first image are the same as theidentified features in the second image, the system and method includes:(a) mapping locations of the identified features in the first image withrespect to the first calibration data, (b) mapping locations of theidentified features in the second image with respect to the secondcalibration data, and (c) computing a transform mapping the mappedfeatures at the second location to the mapped features at the firstlocation. Alternatively, where some of the identified features in thefirst image differ from the identified features in the second image, thesystem and method includes: (a) mapping locations of the identifiedfeatures in the first image with respect to the first calibration data,(b) computing a transform relative to a stored specification of featurelocations of the first workpiece, (c) mapping locations of theidentified features in the second image with respect to the secondcalibration data, (d) using the transform computed in step (b) to derivelocations of the identified features from the second image in the firstcoordinate space when the workpiece is located at the first location,and (e) computing a transform mapping the mapped features at the secondlocation to the corresponding transformed features at the firstlocation. The specification of the first workpiece can be based upon aCAD model of the first workpiece or a measured model (e.g. CMM-generatedmeasurements) of the first workpiece. Illustratively, the system andmethod can include: (a) moving the first workpiece iteratively with amotion rendering device at either the first location or the secondlocation to a plurality of different poses, (b) identifying features ateach of the poses at each of the first location and the second locationand (c) accumulating the identified feature information to enhanceaccuracy, wherein the first workpiece is either the same workpiece or isone of a plurality of discrete workpieces. In various embodiments, thesystem and method includes a mapping from an image coordinate system toa calibration coordinate system at the first location, and wherein themapping is unity. In embodiments, the second location has a secondworkpiece into which the first workpiece is placed into engagement in adesired alignment with the second workpiece, and/or the second workpiececan be a part, a container or a framework for further processing of thefirst workpiece. Additionally, in various embodiments, the operation canbe at least one of an alignment operation with respect to anotherobject, a printing operation on the first workpiece, and an applicationoperation on the first workpiece, and the operation can be performed atleast in part at a location remote from the first location and thesecond location.

In another illustrative embodiment, a system and method for calibratinga vision system in an environment in which a first workpiece at a firstlocation is transferred by a manipulator to a second location, whereinan operation performed on the first workpiece relies upon tying togethercoordinate spaces of the first location and the second location isprovided, and in which at least one of the locations is subject tohand-eye calibration. At least one vision system camera is arranged toimage the first workpiece at the first location and to image the secondlocation. The vision system camera is hand-eye calibrated with respectto the first location to derive first calibration data, and the firstworkpiece is positioned at the first location. Illustratively, the firstworkpiece is moved by the manipulator from the first location to thesecond location, and an image is acquired, from which features on thefirst workpiece are located. The first workpiece is then moved by themanipulator to the first location from the second location, and a poseof the first workpiece is changed at the first location by moving themotion rendering device to a new known pose. The motion rendering devicecan be located at either location and the pose-change via the motionrendering device can occur either before or after movement by themanipulator from the second to first location.

The above steps are iterated until feature location and other datarelevant to hand-eye calibration is accumulated and stored, and then theaccumulated data is used to hand-eye calibrate at least one visionsystem camera with respect to the second location. This allows tyingtogether the first coordinate space and the second coordinate space bythe common coordinate space relative to the motion rendering deviceobtained from the hand-eye calibration. Illustratively, the secondlocation has a second workpiece on the motion rendering device intowhich the first workpiece is placed into engagement in a desiredalignment with the second workpiece. The second workpiece can be a part,a container or a framework for further processing of the firstworkpiece, and the operation can be at least one of an alignmentoperation with respect to another object, a printing operation on thefirst workpiece, and an application operation on the first workpiece.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention description below refers to the accompanying drawings, ofwhich:

FIG. 1 is a diagram of a multi-camera vision system arranged withrespect to an exemplary pick-and-place assembly environment, showing apick-and-place manipulator located at the pick location to extract aworkpiece for transport to a place location, in which one of thelocations includes an exemplary motion stage used in conjunction with aworkpiece assembly system, and a vision processor with associated toolsand processes for calibration, training and runtime operation;

FIG. 2 is a diagram of a two-camera vision system arranged with respectto an exemplary pick-and-place assembly arrangement, in which aworkpiece is in alignment with another workpiece, showing thepick-and-place manipulator located at the place location, and placingthe first, picked workpiece on the second workpiece;

FIG. 3 is a flow diagram showing a generalized overview of a method fortying a first coordinate space at a first location to a secondcoordinate space at a second location based upon imaging of a runtimeworkpiece gripped by the manipulator at each location, in which thecamera system is calibrated with respect to one of the locations;

FIG. 4 is a more detailed flow diagram of the method generally describedin FIG. 3 for tying the coordinate space of a first location to acoordinate space of a second location using features of runtimeworkpiece image(s) at each location, in which the features at eachlocation are the same;

FIG. 5 is a more detailed flow diagram of the method generally describedin FIG. 3 for tying the coordinate space of a first location to acoordinate space of a second location using features of runtimeworkpiece image(s) at each location, in which at least some of thefeatures at each location are different due to visibility differences ateach respective field of view;

FIG. 6 is a flow diagram of a method for tying the coordinate space of afirst location to a coordinate space of a second location by performingseparate hand-eye calibrations at the first location and the secondlocation; and

FIG. 7 is a flow diagram of an additional process relative to themethods of FIGS. 4 and 5 in which features are extracted at multipleunknown poses and the accumulated features are used to tie coordinatespaces at each location.

DETAILED DESCRIPTION

FIG. 1 illustrates an assembly system 100 with an exemplarypick-and-place (manipulator) mechanism 120 that can employ the systemand method for calibrating using an object (first workpiece) that ismanipulated (transferred between locations/stations) by the manipulatorin runtime operation for assembly to a second workpiece at a secondlocation, according to an embodiment. The locations particularly includea pick location 122, where the runtime workpiece-to-be-transferred 124is initially located and a place location 110, where a second workpiece112, to which the first workpiece 124 is assembled, is located. Themanipulator can include a gripper that can comprise (e.g.) a suction cup125 for selectively engaging the first workpiece 124. The manipulator120 selectively picks up (arrow 130) and transfers (arrow 132) the firstworkpiece 124 to an alignment position 126 on the second workpiece 112.In this example, a rail 128 is provided to guide the manipulator 120along a repeatable path. This rail is optional, and any modality thatallows for repeatable motion of the manipulator 120 between pick andplace locations is expressly contemplated. By way of illustration, themanipulator defines a coordinate space 134 characterized by axes Xm, Ymand Zm in which the manipulator 120 moves along at least the axis Ym,and along a pick/place direction in the axis Zm. Rotation ⊖zm about theaxis Zm is also shown, but may or may not be performed by themanipulator. The motions performed by the manipulator are arbitrary andthe vision system generally operates independently of such motions,which are shown by way of illustration.

The pick location 122 and/or place location 110 each define a platformonto which respective workpieces 124 and 112 are positioned prior to apick/place operation. The workpieces 124, 112 can be deposited on thelocations/platforms (122, 110, respectively) by any acceptabletechnique—e.g. a conveyor from a source of workpieces, robotmanipulator, manual placement by a user, etc. In other embodiments, theworkpiece 124 can be held by the manipulator at the first location 110,and is not deposited on the location/platform 122. In the exemplarysystem, Either one of the locations/platforms 122, 110 can comprise amotion stage that moves in one or more degrees of freedom with desiredaccuracy. Such stage motion is provided to establish alignment betweenworkpieces prior to or after the repeatable pick and place motion. Thatis, one of the workpieces is pre-aligned by the motion stage, and thenthe pick/place operation maintains the predetermined alignment as itmoves through the defined path. Alternatively, after the pick/placeoperation moves the workpiece, the final alignment of the workpieces canoccur just prior to the assembly/place motion. Each platform (either ofwhich can include a motion stage) defines its own coordinate space,which can be considered the location's local, calibrated coordinatespace. That is, the pick platform 122 defines a first coordinate spaceXs1, Ys1, Zs1 (orthogonal axes 135), and the place platform 110 definesa second coordinate space Xs2, Ys2, Zs2 (orthogonal axes 137). Whereeither platform includes a motion stage, such motion can occur along oneor more of the depicted coordinate axes and, optionally, in at least onerotational degree of freedom (i.e. along ⊖zs1 or ⊖zs2 as depicted).

In this exemplary system environment/arrangement 100, at least onecamera 142, 140 respectively images each location (122, 110).Alternatively, a single camera or multiple cameras can image bothlocations in a single field of view (FOV). In general, and as describedherein, it is contemplated that the same or different features of theworkpiece are visible to one or more cameras at each location as theworkpiece is positioned at that location. The cameras 140, 142 areinterconnected to a vision process (or) 160. One or both locations 122,110 can optionally include additional cameras 144 (shown in phantom).The vision process(or) 160 is also operatively interconnected to a stagemotion controller 170 at the associated location that provide motioninformation (e.g. encoder or stepper distance/pulse information) 172 tothe processor 160 for use in determining physical position of the stageand associated workpiece. Since the motion stage can be locatedoptionally at either location 122 or 110, the controller 170 andassociated motion information 172 is depicted similarly for bothlocations. During hand-eye calibration (described further below), thestage is moved while motion information is recorded and correlated withthe acquired image(s).

The cameras 140, 142, 144 are each operatively connected to the visionsystem processor and associated vision system process 160 that can beintegrated as a custom vision processor circuit within one or morecamera housing(s), in whole or in part, or can be provided within aninterconnected, computing device 180, including but not limited to, aPC, laptop, tablet, smartphone, or the like having an appropriategraphical user interface (GUI—e.g. display and/or touchscreen 182,keyboard 184, and/or mouse 186) to assist with setup (calibration),training, and/or runtime operation. Note that in assembly systemsemploying more than one camera, each camera is adapted to send eitherits acquired image or information extracted from that image to a centralprocessor. The central processor then integrates the information fromthe various cameras during assembly. The vision process(or) 160 performsa variety of vision system processes (or elements/modules) includingvarious vision tools 162, such as edge finders, blob analyzers, searchtools, caliper tools, etc. Illustratively, the vision process(or) 160includes an alignment process(or) 164 that carries out alignment ofimage data from the two workpieces in a manner described further below.A calibration process(or) 166 facilitates performing camera calibrationsand hand-eye calibrations further described below. Illustratively, atraining process(or) 168 carries out the various training procedurescontemplated herein to reposition the workpieces so as to accuratelyassemble the second workpiece relative to the first workpiece. Note thatthe vision processor 160 can be established as a plurality ofinterconnected camera processors (or other devices), or a centralprocessor in a single camera assembly (or remote computing device).

Note also that the physical workpiece assembly arrangement depicted invarious embodiments herein contains an arbitrary number of cameras thatimage various regions of the platform/motion stage. The number ofcameras used in imaging each location (and/or the overall assemblyenvironment) is highly variable in alternate arrangements. Likewise, thenumber of locations within the overall system at which the manipulatorperforms a task is highly variable.

In addition, it should be clear that the depicted assembly environmentis exemplary of a variety of arrangements in which a first workpiece istransferred by a (e.g.) repeatable manipulator from a first location toanother location in which an operation is performed upon the firstworkpiece. The operation can include engagement with a second workpiecein a desired alignment or can be performed directly upon the firstworkpiece using an appropriate mechanism. For example, a secondworkpiece can be a part, to which the first workpiece is assembled, acontainer/box into which the first workpiece is placed and/or aframework into which the first workpiece is places—for example as partof a kiting arrangement. The in addition to such placement, theoperation can also include printing or applying decals to the firstworkpiece, exposing it to a laser, cutter, tool head or other deviceand/or any other process that modifies the workpiece. Furtherdefinitions for the operation and second workpiece are provided below.In general, it is contemplated mainly that the system and method allowsfor tying the coordinate spaces of the camera(s) imaging the firstlocation and the second location together to enable an operation tooccur in a desirable manner.

Before describing further the details of the illustrative system andmethod, reference is made to the following definitions, which shouldassist the reader in understanding the concepts presented herein:

Definitions “Calibrated Coordinate A coordinate space defined by thecalibration Space” target used during a camera calibration, hand- eyecalibration, or other calibration process. “Common Coordinate Atruntime, features acquired by the cameras in Space” the system aremapped to this space. The common coordinate space is shared acrosslocations. “Image Coordinate The coordinate space of an acquired imageor Space” the coordinate space of a transformed image. “MotionCoordinate The native coordinate space associated with the Space” motionrendering device (e.g. motion stage). “Workpiece Coordinate Thecoordinate space associated with a Space” workpiece. Possible sources ofthis coordinate space are a CAD specification or CMM rendition of theworkpiece. “CalibratedFromImage The transform that maps points from theImage Transform” Coordinate Space to the Calibrated Coordinate Space.“Camera Calibration” A process to establish the transform between theImage and Calibrated Coordinate Spaces. “Hand-eye Calibration” Aprocess, know to those of skill, to establish the transforms between theImage, the Calibrated, and the Motion Coordinate Spaces. “all” camerasRefers to all cameras that are used by the system in the assembly taskherein. It is expressly contemplated that some cameras used by thevision system (or other processes), which may otherwise image the scene,can be omitted from the assembly task. It is contemplated mainly thatone or more (at least one) camera(s) (which can be the same camera)image each location in the assembly and is/are calibrated to the commoncoordinate system space. “operation” Refers to a process performed on orwith respect to the first workpiece at the first location, secondlocation or a location at least partially remote from the first locationand the second location. The operation can be (e.g.) an assemblyoperation with respect to a second workpiece, placing the firstworkpiece in a box or framework (i.e. kitting), or a modification to thefirst workpiece, such as printing, decal application, adhesiveapplication, etc., using appropriate mechanisms “first workpiece” Refersto a part or other object used in actual runtime operation of themanipulator system (e.g. an assembly system)-also termed a “runtimeworkpiece”. The first workpiece expressly excludes a calibrationtarget/plate or other object used for system setup/training that is notpart of a manufacturing or other runtime process, post-calibration andtraining. “second workpiece” Refers to a workpiece located at a secondlocation to which the first workpiece is assembled based upon analignment that can be achieved (e.g.) using a motion rendering device(motion stage) at either the first location or the second location. Asecond workpiece can also refer to a container (box) or framework intowhich the first workpiece is placed.

With reference briefly to FIG. 2, the manipulator 120 is shown havingmoved along the rail 180 from the first location 122 to the secondlocation 110, with the first workpiece 124 in its gripper 150. Themanipulator 120 is depicted accurately placing (arrow 250) the firstworkpiece 126 into the aligned receiving location 126 on the secondworkpiece 112. This alignment is established by the cameras 140, 142 andassociated vision system process(or) 160 by moving the motion stagebased on the common coordinate space. This common coordinate space isestablished during calibration by tying the calibrated coordinate spaceof each of the two locations 122, 110 together as now described below.

By way of a general understanding of certain calibration principles, fora rigid body, such as a calibration target, a motion can becharacterized by a pair of poses: a starting pose immediately precedinga motion, and an ending pose immediately following the motion—a “pose”herein being defined as a set of numerical values to describe theposition and orientation of a body, at any one particular instant intime, in some underlying coordinate space—a virtual characterization ofthe body. For example, in two dimensions, a pose can be characterized bythree numbers: a translation in X, a translation in Y, and a rotation R(or ⊖). A pose in the context of a calibration target describes how thecalibration target is presented to the camera(s),. Typically, in astandard so-called “hand-eye calibration”, a calibration target is movedby a motion rendering device to a number of different poses with respectto the camera(s), and each camera acquires an image of the calibrationtarget at each such pose. The goal of such hand-eye calibration is todetermine the poses of the camera(s), the poses of the calibrationtarget and the poses of the motion rendering device in a singlecoordinate space, which can be termed the “calibrated coordinate space”.Typically, “motion” is provided by a physical device that can renderphysical motion, such as a robot arm, or a motion stage, or a gantry.Note that either the target can move relative to one or more stationarycamera(s) or the camera(s) can move relative to a stationary target,such as when the cameras are mounted to the physical device providingmotion. The controller of such a motion-rendering device employsnumerical values (i.e. poses) to command the device to render anydesired motion, and those values are interpreted in a native coordinatespace for that device, termed herein the “motion coordinate space”. Withreference now to FIG. 3, an overall calibration process 300 for tyingthe coordinate spaces of two discrete locations (e.g. a pick and a placelocation) using a runtime workpiece moved by the manipulator between thelocations is described. Notably, the use of a runtime workpiece avoidsthe disadvantages associated with use of a calibration target, andallows the user to conveniently employ the same structure that themanipulator is expected to handle in runtime operation. The process 300begins in step 310 by arranging one or more cameras to image/acquireimage(s) of a first location and a second location. The camera(s) viewfeature(s) of a runtime workpiece at each location—in which the featurescan be all, or in part, the same at each location, or different at eachlocation. In step 320, the camera(s) viewing features at the firstlocation or the second location are calibrated to a respectivecalibrated coordinate space. This can comprise hand-eye calibration bymoving a calibration target by the stage and establishing the coordinatespace based on such motion in association with motion informationprovided by the stage. Alternatively, this can comprise a calibrationprocess that can be achieved using image(s) of a special calibrationtarget such as the checker grid calibration plate, thus tying all thecameras at the location to the respective coordinate space. In step 330,images of the runtime workpiece are acquired by one or more camera(s)viewing the workpiece at the first location and the locations offeatures on the runtime workpiece are identified/extracted at the firstlocation. The identified/extracted features are then used to associatethe workpiece with respect to a first calibrated coordinatespace—typically related to the first location in step 340. The featurescan be any identifiable visual elements on the workpiece, and inparticular those that alone, or in combination uniquely define theorientation and location of the workpiece. Thus, where edges and cornersare employed as identified features, such edges and corners can beunique to a certain location on the overall workpiece (e.g. a notch onone side of the workpiece). Feature extraction can be accomplished usingvarious vision system tools (e.g. edge detectors, blob analyzers, etc.)162 (FIG. 1) in accordance with skill in the art.

In step 350, the runtime workpiece is gripped by the manipulator (e.g.by application of a suction cup 125 to the runtime workpiece 124 in FIG.1), and the workpiece is moved to the second location. This motionoccurs at least one time (and can occur multiple times using differentposes as described with reference to FIG. 7 below). Then, in step 360,the camera(s) at (viewing workpiece features in) the second locationacquire(s) one or more images of the workpiece (in the image coordinatespace, thereof). Features that are the same as those viewed at the firstlocation, or that differ are used at the second location to findlocations on the workpiece at the second location. Where the featuresare the same, the two coordinate spaces can be directly tied. Where thefeatures different, a specification of the workpiece, such as a CADdrawing or measured representation/rendition (i.e. using a coordinatemeasuring machine CMM to measure the workpiece and store dimensions)allows the features at each location to be corresponded by the visionsystem calibration process. The corresponded features then allow eachcoordinate space to be tied together. Note that where one of thelocations is hand-eye calibrated via step 320, the computed commoncoordinate space can be the same as the motion coordinate space of themotion stage.

FIGS. 4 and 5 are more detailed descriptions of two discretemethods/processes 400 and 500, respectively, for tying coordinate spacestogether at the first location and the second location in accordancewith the generalized process of FIG. 3. In FIG. 4, the process 400includes step 410 in which one or more cameras are arranged to image twolocations, one of which can include a motion stage. As described above,optional hand-eye calibration can be performed with the camera(s)viewing the location with a motion stage to establish the calibratedcoordinate space. More generally, in steps 420 and 430, the camera(s)imaging the first and second locations can be calibrated using anappropriate calibration target. In step 440, the camera(s) acquire oneor more images of the workpiece at the first location and the images areused to locate features, featuresImage1, on the workpiece usingappropriate vision system tools (step 450). These features can be mappedto the calibrated coordinate space at the first location(featuresCalibrated1=Calibrated1FromImage1*featuresImage1). The runtimeworkpiece is then gripped and moved by the manipulator to the secondlocation. Where the camera(s) acquire image(s) of the runtime object(step 460). In this technique, the same features, featuresImage1 arealso visible by the camera(s) at the second location, featuresImage2. Instep 470, the camera(s) at the second location, thus, locates the samefeatures on the runtime workpiece and the process 400 maps the locationsbased on the calibration data at the second location(featuresCalibrated2=Calibrated2FromImage2*featuresImage2 ). Then, instep 480, the process 400 computes a transform,Calibrated1FromCalibrated2, mapping featuresCalibrated2 to thefeaturesCalibrated1 (from step 450). This transform is used to tie thecoordinate spaces at the two locations together in accordance with thefollowing relationship:

featuresCalibrated1=Calibrated1FromCalibrated2*featuresCalibrated2

If the location containing the stage has been optionally hand-eyecalibrated, then the transform Calibrated1FromCalibrated2 can be used incombination with the hand-eye calibration results to guide the assemblyof the part during runtime.

In FIG. 5, the process 500 includes step 510 in which one or morecameras are arranged to image two locations, one of which can include amotion stage. As described above, optional hand-eye calibration can beperformed with the camera(s) viewing the location with a motion stage toestablish the calibrated coordinate space. In steps 520 and 530, thefirst and second locations can be calibrated using an appropriatecalibration target. In step 540, the camera(s) acquire one or moreimages of the workpiece at the first location and the images are used tolocate features on the workpiece, featuresImage1, using appropriatevision system tools. In step 550, these features can be mapped to thecalibrated coordinate space at the first location, featuresCalibrated1.In step 560 the process 500 computes a transform,WorkpieceFromCalibrated1, that maps workpiece features in the firstcalibrated coordinate space, featuresCalibrated1 (from step 550) at thefirst location to the stored workpiece coordinate space, in accordancewith the following relationship:

featuresWorkpiece=WorkpieceFromCalibrated1*featuresCalibrated1

The workpiece coordinate space is established based upon a computeraided design (CAD) model of the workpiece that includes a representationof the feature parameters. Alternatively, the specification of theruntime workpiece coordinate space can be established by physicalmeasurement of the workpiece—for example using a coordinate measuringmachine (CMM) in accordance with ordinary skill. Parameters are storedfor use by the mapping process.

The runtime workpiece is then gripped and moved by the manipulator tothe second location. Where the camera(s) acquire image(s) of the runtimeobject (step 570). In this technique, one or more of the features viewedat the second location could differ from those features viewed/imaged atthe first location. This can result when the same features are notvisible to the camera(s) at both locations due to obstructions, thefield of view (FOV) of the camera(s), etc. In step 580, the camera(s) atthe second location, locate the visible features, featVisIn2Image2, onthe runtime workpiece and the locations are mapped(featVisIn2Calibrated2=Calibrated2FromImage2*featVisIn2Image2) based onthe calibration data at the second location. The corresponding points inthe workpiece coordinate space are found, featVisIn2Workpiece Then, instep 590, the process 500 uses the inverse transformWorkpieceFromCalibrated1 from step 560 to compute the location offeatures visible at the second location in the first calibrated spacewhen such features were at the first location:

featVisIn2Calibrated1=Calibrated1FromWorkpiece*featVisIn2Workpiece.

In step 592, the computed feature locations from the first location andthe corresponding detected feature locations at the second location areused to tie the coordinate spaces together at each of the locations inaccordance with the following relationship:

featVisIn2Calibrated1=Calibrated1FromCalibrated2*featVisIn2Calibrated2

If the location containing the stage has been hand-eye calibrated, thenthe transform Calibrated1FromCalibrated2 can be used in combination withthe hand-eye calibration results to guide the assembly of the partduring runtime.

Note, in various embodiments, it is expressly contemplated that themapping from the image coordinate space to calibration coordinate spaceat the first location can be equal to unity. This technique accommodatesarrangements in which the second location is calibrated, and the imagefeatures at the first location are mapped to the calibration coordinatespace at the second location as described in FIGS. 3-5.

FIG. 6 details a method/process 600 according to anther contemplatedtechnique for tying the coordinate spaces of two discrete locationstogether, in which both the first location camera(s) and the secondlocation camera(s) are free of calibration. In this arrangement it iscontemplated that each location can be hand-eye calibrated separately,both locations being tied to the same common coordinate space relativeto the motion stage. In step 610, the camera(s) are arranged to imageeach location. The location (i.e. first location) containing the motionstage is hand-eye calibrated in step 620. The runtime workpiece is thenplaced at the stage-containing first location in step 630. The runtimeworkpiece is then gripped and moved to the second location in step 640,where one or more images are acquired with the camera(s). Features onthe workpiece are located in step 650. Optionally if the CAD or CMMspecification of the workpiece is available, such features can becorresponded with respect to a CAD or CMM representation of theworkpiece coordinate space. The pose derived from the features is storedin a list X11. The workpiece is again gripped and moved back to thefirst/stage location in step 660. The steps 640, 650 and 660 arerepeated as the stage is moved to establish new poses with eachiteration (step 680) until sufficient data is captured and stored in thelist X11. The process 600, then proceeds (via decision step 670) in step690 to use the accumulated data from step 650 (list X11) to hand-eyecalibrate the camera(s) at the second location. The camera calibratedcoordinate spaces at both locations are, thus, tied together by thecommon coordinate space relative to the motion stage.

The methods/processes 400 and 500 (steps 450, 470 in FIG. 4 and steps550 and 580 in FIG. 5) can be enhanced by performing feature extractionfrom the runtime workpiece iteratively. An overview of this enhancedprocess 700 is described in FIG. 7. In step 710, one or more images areacquired of the runtime workpiece at the first location, and featuresare extracted/identified. The workpiece is then gripped and moved by themanipulator in step 720, and features are extracted/identified (eitherthe same or different features as described above). The manipulatoreither returns the workpiece to the first location iteratively, roughlychanging the pose each time (step 730) or transfers multiple instancesof the workpiece with a roughly different pose. That is, the accuracyand robustness of the calibration can be enhanced through multipleiterations of feature extraction and accumulation on either the sameworkpiece of a series of different workpieces that are each run througha motion cycle between locations. During each iteration, features fromimages at the first location and the second location are extracted viasteps 710 and 720 until sufficient feature data is obtained. Thisthreshold can be based on a set number of iterations or another metric.The process 700 then proceeds (via decision step 740) to step 750 wherethe accumulated feature data from multiple poses is used to tie thecoordinate spaces at each location together as described generallyabove.

It should be clear that the above-described techniques for tying thecoordinate spaces at two discrete locations in an assembly processtogether using a runtime workpiece avoids disadvantages associated withuse of a calibration target. These techniques allow for flexibility inthe manner in which cameras are arranged with respect to each location.These techniques also allow for enhanced/refined accuracy throughiteration of various steps, such as feature extraction. These techniquesalso avoid the disadvantages of the above-described prior techniques,which rely on either a known mechanical arrangement or an iterativetrial and error approach.

The foregoing has been a detailed description of illustrativeembodiments of the invention. Various modifications and additions can bemade without departing from the spirit and scope of this invention.Features of each of the various embodiments described above may becombined with features of other described embodiments as appropriate inorder to provide a multiplicity of feature combinations in associatednew embodiments. Furthermore, while the foregoing describes a number ofseparate embodiments of the apparatus and method of the presentinvention, what has been described herein is merely illustrative of theapplication of the principles of the present invention. For example, asused herein the terms “process” and/or “processor” should be takenbroadly to include a variety of electronic hardware and/or softwarebased functions and components (and can alternatively be termedfunctional “modules” or “elements”). Moreover, a depicted process orprocessor can be combined with other processes and/or processors ordivided into various sub-processes or processors. Such sub-processesand/or sub-processors can be variously combined according to embodimentsherein. Likewise, it is expressly contemplated that any function,process and/or processor herein can be implemented using electronichardware, software consisting of a non-transitory computer-readablemedium of program instructions, or a combination of hardware andsoftware. Additionally, as used herein various directional andorientational terms such as “vertical”, “horizontal”, “up”, “down”,“bottom”, “top”, “side”, “front”, “rear”, “left”, “right”, and the like,are used only as relative conventions and not as absolute orientationswith respect to a fixed coordinate space or system, such as the actingdirection of gravity. Illustratively, one location includes a locationstage, but it is contemplated that multiple locations can includestages—for example where a first motion stage moves the workpiece alongone axis and the second stage moves the workpiece along anotherorthogonal axis (or a rotation not provided by the first stage).Accordingly, this description is meant to be taken only by way ofexample, and not to otherwise limit the scope of this invention.

What is claimed is:
 1. A method for calibrating a vision system in anenvironment in which a first workpiece at a first location istransferred by a manipulator to a second location, wherein an operationperformed on the first workpiece relies upon tying together coordinatespaces of the first location and the second location, the methodcomprising the steps of: arranging at least one vision system camera toimage the first workpiece when positioned at the first location and toimage the first workpiece when positioned at the second location;calibrating at least one vision system camera with respect to the firstlocation to derive first calibration data which defines a firstcoordinate space and at least one vision system camera with respect tothe second location to derive second calibration data which defines asecond coordinate space; identifying features of at least the firstworkpiece at the first location from a first image of the firstworkpiece; based on the identified features in the first image locatingthe first workpiece with respect to the first coordinate space relativeto the first location; gripping and moving, with the manipulator, atleast one time, the first workpiece to a predetermined manipulatorposition at the second location; acquiring a second image of the firstworkpiece at the second location; and based upon the identified featuresin the second image, locating the first workpiece with respect to thesecond coordinate space relative to the second location and tyingtogether the first coordinate space and the second coordinate space. 2.The method as set forth in claim 1 wherein the identified features inthe first image are the same as the identified features in the secondimage, and further comprising (a) mapping locations of the identifiedfeatures in the first image with respect to the first calibration data,(b) mapping locations of the identified features in the second imagewith respect to the second calibration data, and (c) computing atransform mapping the mapped features at the second location to themapped features at the first location.
 3. The method as set forth inclaim 1 wherein some of the identified features in the first imagediffer from the identified features in the second image, and furthercomprising (a) mapping locations of the identified features in the firstimage with respect to the first calibration data, (b) computing atransform relative to a stored specification of feature locations of thefirst workpiece, (c) mapping locations of the identified features in thesecond image with respect to the second calibration data, (d) using thetransform computed in step (b) to derive locations of the identifiedfeatures from the second image in the first coordinate space when theworkpiece is located at the first location, and (e) computing atransform mapping the mapped features at the second location to thecorresponding transformed features at the first location.
 4. The methodas set forth in claim 3 wherein the specification is based upon a CADmodel of the first workpiece.
 5. The method as set forth in claim 3wherein the specification is based upon a measured model of the firstworkpiece.
 6. The method as set forth in claim 1 further comprising (a)moving the first workpiece iteratively with a motion rendering device ateither the first location or the second location to a plurality ofdifferent poses, (b) identifying features at each of the poses at eachof the first location and the second location and (c) accumulating theidentified feature information to enhance accuracy, wherein the firstworkpiece is either the same workpiece or is one of a plurality ofdiscrete workpieces.
 7. The method as set forth in claim 1 furthercomprising providing a mapping from an image coordinate system to acalibration coordinate system at the first location, and wherein themapping is unity.
 8. The method as set forth in claim 1 wherein thesecond location has a second workpiece into which the first workpiece isplaced into engagement in a desired alignment with the second workpiece.9. The method as set forth in claim 8 wherein the second workpiece is apart, a container or a framework for further processing of the firstworkpiece.
 10. The method as set forth in claim 1 wherein the operationis at least one of an alignment operation with respect to anotherobject, a printing operation on the first workpiece, and an applicationoperation on the first workpiece.
 11. The method as set forth in claim10 wherein the operation is performed at least in part at a locationremote from the first location and the second location.
 12. A method forcalibrating a vision system in an environment in which a first workpieceat a first location is transferred by a manipulator to a secondlocation, wherein an operation performed on the first workpiece reliesupon tying together coordinate spaces of the first location and thesecond location, the method comprising the steps of: (a) arranging atleast one vision system camera to image the first workpiece at the firstlocation and to image the second location; (b) hand-eye calibrating atleast one vision system camera with respect to the first location toderive first calibration data; (c) positioning the first workpiece atthe first location; (d) moving the first workpiece from the firstlocation to the second location; (e) acquiring an image and locatingfeatures on the first workpiece; (f) moving the first workpiece to thefirst location from the second location and changing a pose of the firstworkpiece at the first location by moving the motion rendering device toa new known pose; (g) iterating steps (d-f) until feature location andother data relevant to hand-eye calibration is accumulated; and (h)using the accumulated data to hand-eye calibrate at least one visionsystem camera with respect to the second location, and tying togetherthe first coordinate space and the second coordinate space by the commoncoordinate space relative to the motion rendering device obtained fromthe hand-eye calibration.
 13. The method as set forth in claim 12wherein the second location has a second workpiece on the motionrendering device into which the first workpiece is placed intoengagement in a desired alignment with the second workpiece.
 14. Themethod as set forth in claim 13 wherein the second workpiece is a part,a container or a framework for further processing of the firstworkpiece.
 15. The method as set forth in claim 12 wherein the operationis at least one of an alignment operation with respect to anotherobject, a printing operation on the first workpiece, and an applicationoperation on the first workpiece.
 16. A system for calibrating a visionsystem in an environment in which a first workpiece at a first locationis transferred by a manipulator to a second location, wherein anoperation performed on the first workpiece relies upon tying togethercoordinate spaces of the first location and the second location,comprising: at least one vision system camera arranged to image thefirst workpiece when positioned at the first location and to image thefirst workpiece when positioned at the second location; a calibrationprocess that calibrates at least one vision system camera with respectto the first location to derive first calibration data and the at leastone vision system camera with respect to the second location to derivesecond calibration data; a feature extraction process that identifiesfeatures of at least the first workpiece at the first location from afirst image of the first workpiece, and based on the identified featuresin the first image, that locates the first workpiece with respect to afirst coordinate space relative to the first location, and based uponthe identified features in a second image at a second location, thatlocates the first workpiece with respect to a second coordinate spacerelative to the second location; and a calibration process that tiestogether the first coordinate space and the second coordinate space. 17.The system as set forth in claim 16 wherein the second location has asecond workpiece into which the first workpiece is placed intoengagement in a desired alignment with the second workpiece.
 18. Thesystem as set forth in claim 16 wherein some of the identified featuresin the first image differ from the identified features in the secondimage, and wherein the calibration process is constructed and arrangedto (a) map locations of the identified features in the first image withrespect to the first calibration data, (b) compute a transform relativeto a stored specification of feature locations of the first workpiece,(c) map locations of the identified features in the second image withrespect to the second calibration data, (d) using the transform computedin step (b) derive locations of the identified features from the secondimage in the first coordinate space when the workpiece is located at thefirst location, and (e) compute a transform mapping the mapped featuresat the second location to the corresponding transformed features at thefirst location.
 19. The system as set forth in claim 18 wherein thespecification is based upon either a CAD model of the first workpiece ora measured model of the first workpiece.
 20. The system as set forth inclaim 16 wherein the second location has a second workpiece into whichthe first workpiece is placed into engagement in a desired alignmentwith the second workpiece.